Intelligent method and system for producing and displaying stereoscopically-multiplexed images of three-dimensional objects for use in realistic stereoscopic viewing thereof in interactive virtual reality display environments

ABSTRACT

An intelligent system and process for producing and displaying stereoscopically-multiplexed images of either real or synthetic 3-D objects, for use in realistic stereoscopic viewing thereof. The system comprises a subsystem for acquiring parameters specifying the viewing process of a viewer positioned relative to a display surface associated with a stereoscopic display subsystem. A computer-based subsystem is provided for producing stereoscopically-multiplexed images of either the real or synthetic 3-D objects, using the acquired parameters. The stereoscopically-multiplexed images are on the display surface, for use in realistic stereoscopic viewing of either the real or synthetic 3-D objects, by the viewer.

RELATED CASES

This patent application is a Continuation-in-Part of patent applicationSer. No. 08/339,986 entitled "Desktop-Based Projection Display SystemFor Stereoscopic Viewing of Displayed Imagery Over A Wide Field Of View"filed Nov. 14, 1994 by Dentinger, et al., now U.S. Pat. No. 5,502,481;patent application Ser. No. 08/126,077 entitled "A System for Producing3-D Stereo Images" filed Sep. 23, 1993 by Sadeg M. Faris, now U.S. Pat.No. 5,537,144; patent application Ser. No. 08/269,202 entitled "Methodsfor Manufacturing Micro-Polarizers" filed on Jun. 30, 1994, abandoned bySadeg M. Faris; and patent application Ser. No. 07/976,518 entitled"Method and Apparatus for Producing and Recording Spatially-MultiplexedImages for Use in 3-D Stereoscopic Viewing Thereof" filed Nov. 16, 1992by Sadeg M. Faris, now U.S. Pat. No. 5,553,203. Each of these patentapplications is incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

1. Field of Invention

The present invention relates to an improved method and system forproducing stereoscopically-multiplexed images from stereoscopicimage-pairs and displaying the same stereoscopically, in an interactivemanner that allows viewers to perceive displayed imagery with a sense ofrealism commensurate with natural viewing of physical reality.

2. Brief Description of State of the Art

In the contemporary period, stereoscopic display systems are widely usedin diverse image display environments, including virtual-realityapplications. The value of such image display systems resides in thefact that viewers can view objects with depth perception inthree-dimensional space.

In general, stereoscopic image display systems display pairs ofstereoscopic images (i.e. stereoscopic image-pairs) to the eyes of humanviewers. In principle, there are two ways in which to producestereoscopic image-pairs for use in stereoscopic display processes. Thefirst technique involves using a "real" stereoscopic-camera, positionedwith respect to a real 3-D object or scene, in order to acquire eachpair of stereoscopic images thereof. The second techniques involvesusing a computer-based 3-D modeling system to implement a "virtual"stereoscopic-camera, positioned with respect to a (geometric) model of a3-D object or scene, both represented within the 3-D modeling system. Inthe first technique, it is necessary to characterize the real-imageacquisition process by specifying the camera-parameters of the realstereoscopic-camera used during the image acquisition process. In thesecond technique, it is necessary to characterize the virtual-imageacquisition process by specifying the "camera-parameters" of the virtualstereoscopic-camera used during the image acquisition process. In eithercase, the particular selection of camera parameters for either the realor virtual stereoscopic-camera necessarily characterizes importantproperties in the stereoscopic image-pairs, which are ultimatelystereoscopically-multiplexed, using one or another format, prior todisplay.

Presently, there are several known techniques for producing"spectrally-multiplexed images", i.e. producing temporal-multiplexing,spatial-multiplexing and spectral-multiplexing.

Presently, there exist a large number of prior art stereoscopic displaysystems which use the first technique described above in order toproduce stereoscopically-multiplexed images for display on the displaysurfaces of such systems. In such prior art systems, the viewer desiresto view stereoscopically, real 3-D objects existing in physical reality.Such systems are useful in laprascopic and endoscopic surgery,telerobotics, and the like. During the stereoscopic display process,complementary stereoscopic-demultiplexing techniques are used in orderto provide to the left and right eyes of the viewer, the left and rightimages in the produced stereoscopic image-pairs, and thus permit theviewer to perceive full depth sensation. However, the selection ofcamera parameters used to produce the displayed stereoscopic image-pairsrarely, if ever, correspond adequately with the "viewing parameters" ofthe viewer's, human vision system, which ultimately views the displayedstereoscopic image-pairs on the display surface before which the viewerresides.

Also, there exist a large number of prior art stereoscopic displaysystems which use the second technique described above in order toproduce stereoscopically-multiplexed images for display on the displaysurfaces of such systems. In such systems, the viewer desires to viewstereoscopically, synthetic 3-D objects existing only in virtualreality. Such systems are useful in flight simulation and training,virtual surgery, video-gaming applications and the like. During thestereoscopic display process, complementary stereoscopic-demultiplexingtechniques are also used to provide to the left and right eyes of theviewer, the left and right images in the produced stereoscopicimage-par. However, the selection of camera parameters used to producethe displayed stereoscopic image-pairs in such systems rarely, if ever,correspond adequately with the viewing parameters of the viewer's humanvision system, which ultimately views the displayed stereoscopicimage-pairs on the display surface before which the viewer resides.

Consequently, stereoscopic viewing of either real or synthetic 3-Dobjects in virtual reality environments, using prior art stereoscopicimage production and display systems, have generally lacked the sense ofrealism otherwise experienced when directly viewing real 3-D scenery orobjects in physical reality environments.

Thus there is a great need in the art for a stereoscopic imageproduction and display system having the functionalities required inhigh performance virtual-reality based applications, while avoiding theshortcomings and drawbacks associated with prior art systems andmethodologies.

OBJECTS AND SUMMARY OF THE INVENTION

Accordingly, it is a primary object of the present invention to providean interactive-based system for producing and displayingstereoscopically-multiplexed images of either real or synthetic 3-Dobjects that permits realistic stereoscopic viewing thereof, whileavoiding the shortcomings and drawbacks of prior art systems andmethodologies.

Another object of the present invention is to provide such a system, inwhich the true viewing parameters of the viewer, including head/eyeposition and orientation, are continuously acquired relative to thedisplay surface of the stereoscopic display subsystem and used duringthe producing of stereoscopically-multiplexed images of synthetic 3-Dobjects being stereoscopically viewed by the viewer in a virtual reality(VR) viewing environment, such as presented in flight simulation andtraining, virtual surgery, video-gaming and like applications.

A further object of the present invention is to provide such a system,in which the true viewing parameters of the viewer, including head/eyeposition and orientation, are continuously acquired relative to thedisplay surface of the stereoscopic display subsystem and used duringthe producing of stereoscopically-multiplexed images of real 3-D objectsbeing stereoscopically viewed by the viewer in a virtual reality (VR)viewing environment, such as presented in laprascopic and endoscopicsurgery, telerobotic and like applications.

Another object of the present invention is to provide such a system, inwhich the stereoscopically-multiplexed images are spatially-mulitplexedimages (SMIs) of either real or synthetic 3-D objects or scenery.

Another object of the present invention is to provide a process forproducing and displaying, in real-time, spatially-mulitplexed images(SMIs) of either real or synthetic 3-D objects or scenery, wherein thetrue viewing parameters of the viewer, including head/eye position andorientation, are continuously acquired relative to the display surfaceof the stereoscopic display subsystem and used during the producing ofstereoscopically-multiplexed images of either the real or synthetic 3-Dobjects being stereoscopically viewed by the viewer in a virtual reality(VR) viewing environment.

Another object of the present invention is to provide a stereoscopiccamera system which is capable of acquiring , on a real-time basis,stereoscopic image-pairs of real 3-D objects and scenery using cameraparameters that correspond to the range of viewing parameters thatcharacterize the stereoscopic vision system of typical human viewers.

Another object of the present invention is to provide a system ofcompact construction, such as notebook computer, for producing anddisplaying, in real-time, micropolarized spatially-mulitplexed images(SMIs) of either real or synthetic 3-D objects or scenery, wherein thetrue viewing parameters of the viewer, including head/eye position andorientation, are continuously acquired relative to the display surfaceof the portable computer system and used during the production ofspatially-multiplexed images of either the real or synthetic 3-D objectsbeing stereoscopically viewed by the viewer in a virtual reality (VR)viewing environment, wearing a pair of electrically-passive polarizingeye-glasses.

Another object of the present invention is to provide such a system inthe form of a desktop computer graphics workstation, particularlyadapted for use in virtual reality applications.

It is yet a further object of the present invention to provide such asystem and method that can be carried out using otherstereoscopic-multiplexing techniques, such as time-sequential (i.e.field-sequential) multiplexing or spectral-multiplexing techniques.

Another object of the present invention is to provide a stereoscopicdisplay system as described above, using either direct or projectionviewing techniques, and which can be easily mounted onto a moveablesupport platform and thus be utilizable in flight-simulators,virtual-reality games and the like.

A further object of the present invention is to provide astereoscopic-multiplexing image production and display system which isparticularly adapted for use in scientific visualization of diverse datasets, involving the interactive exploration of the visual nature andcharacter thereof.

These and other objects of the present invention will become apparenthereinafter and in the claims to invention.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the Objects of the PresentInvention, the Detailed Description of the Illustrated Embodimentsshould be read in conjunction with the accompanying Drawings, in which:

FIG. 1 is a perspective view of the interactive-based,stereoscopically-multiplexed image production and display system of thepresent invention, showing the various subsystems, subcomponents, andcoordinate reference systems embedded therein, in relation to each otherand a viewer wearing a pair of electrically-passive polarizationspectacles in front of his or her eyes, and being free to move withrespect to the LCD display surface of the stereoscopic image-pairdisplay subsystem;

FIG. 2 is a block system diagram of the stereoscopically-multiplexedimage production and display system of the present invention, showingthe stereoscopic image-pair production subsystem, eye/head position andorientation tracking subsystem, the display surface position andorientation tracking subsystem, the stereoscopic image-multiplexingsubsystem, and the stereoscopic image-pair display subsystem thereof;

FIG. 2A is a block functional diagram of thestereoscopically-multiplexed image generation subsystem of thestereoscopically-multiplexed image production subsystem of the presentinvention;

FIG. 2B is a schematic representation of thestereoscopically-multiplexed image acquisition subsystem of thestereoscopically-multiplexed image production subsystem of the presentinvention;

FIG. 2C is a block functional diagram of the eye/head position andorientation tracking subsystem of the present invention;

FIG. 2D is a block functional diagram of thestereoscopically-multiplexed image display subsystem of the presentinvention;

FIG. 2E is a block functional diagram of the display-surface positionand orientation tracking subsystem of the present invention;

FIG. 3A is a schematic representation of the generalizedstereoscopically-multiplexed image production, display and viewingprocess of the present invention, graphically illustrating the variousprojection and display surfaces, coordinate reference systems,transformations and mappings utilized in the process, wherein the 3-Dobject or scenery may exist either in physical reality, or in virtualreality represented within the stereoscopically-multiplexed imagegeneration subsystem of the present invention;

FIG. 3B is a schematic representation of the generalizedstereoscopically-multiplexed image production, display and viewingprocess of FIG. 3A, setting forth the various projection and displaysurfaces, coordinate reference systems, transformations, mappings andparameters utilized by the particular subsystems at each stage of theprocess;

FIG. 4A is a schematic representation of the subprocess of mapping aprojected perspective image geometrically represented on a continuousprojection surface (Sc), to quantized perspective image represented as apixelized image representation on a quantized projection surface (Sp),carried out within the stereoscopically-multiplexed image productionsubsystem of the present invention;

FIG. 4B is a schematic representation showing, in greater detail, thequantized perspective image mapped on quantized projection surface (Sp)in the form of a pixelized image representation, during the mappingsubprocess of FIG. 4A;

FIG. 4C is a schematic representation showing, in yet greater detail,the use of a kernel function during the stereoscopically-multiplexedimage process of the present invention;

FIG. 5A is a schematic representation of the spatially-multiplexed image(SMI) production, display and viewing process of the present inventionbased on spatial-multiplexing principles, graphically illustrating thevarious projection and display surfaces, coordinate reference systems,transformations and mappings utilized in the process, wherein the 3-Dobject or scenery exists in virtual reality represented within thestereoscopically-multiplexed image generation subsystem of the presentinvention;

FIG. 5B is a schematic representation of thestereoscopically-multiplexed image production, display and viewingprocess of FIG. 5A, setting forth the various projection and displaysurfaces, coordinate reference systems, transformations, mappings andparameters utilized by the particular subsystems at each stage of theprocess;

FIG. 6A is a schematic representation of the spatially-multiplexed image(SMI) production, display and viewing process of the present invention,graphically illustrating the various projection and display surfaces,coordinate reference systems, transformations and mappings utilized inthe process, wherein the 3-D object or scenery exists in physicalreality;

FIG. 6B is a schematic representation of the spatially-multiplexed imageproduction, display and viewing process of FIG. 6A, setting forth thevarious projection and display surfaces, coordinate reference systems,transformations, mappings and parameters utilized by the particularsubsystems at each stage of the process; and

DETAILED DESCRIPTION OF THE ILLUSTRATED EMBODIMENTS

Referring to FIGS. 1 through 2E, the apparatus of the present inventionwill be described in great detail hereinbelow. In the illustrativeembodiment, the apparatus of the present invention is realized in theform of an interactive-based stereoscopically-multiplexed imageproduction and display system. It is understood, however, that thepresent embodiment may be embodied in other systems without departingfrom the scope and spirit of the present invention.

The system and method of the present invention may utilize any one or anumber of available stereoscopically-multiplexing techniques, such astemporal-multiplexing (i.e. multiplexing), spatial-multiplexing orspectral-multiplexing. For purposes of illustration,spatial-multiplexing will be described. It is understood, that whenusing other stereoscopic display techniques to practice the system andmethod of the present invention, various modifications will need to bemade. However, after having read the teachings of the presentdisclosure, such modifications will be within the knowledge of one ofordinary skill in the art.

As shown in FIGS. 1 and 2, the system of the present invention,indicated by reference numeral; generally comprises several majorcomponents, namely: stereoscopic image-pair production subsystem 2;eye/head position and orientation tracking subsystem 3; display surface(panel) position and orientation tracking subsystem 4;stereoscopically-multiplexed image production subsystem 5; and thestereoscopically-multiplexed image display subsystem 6. As shown in FIG.1 and FIGS. 2C-2E, eye/head position and orientation tracking subsystem3, display surface (panel) position and orientation tracking subsystem4, and the stereoscopically-multiplexed image display subsystem 6cooperate together to provide an interactive-basedstereoscopically-multiplexed image display subsystem. The variousfunctionalities of these subsystems will be described in great detailbelow.

It is understood there will be various embodiments of the system of thepresent invention depending on whether stereoscopic image-pairs(comprising pixels selected from left and right perspective images) areto be produced from either (i) real 3-D objects or scenery existing inphysical reality or (ii) virtual (synthetic) 3-D objects or sceneryexisting in virtuality. In either event, the real or synthetic 3-Dobject will be referenced to a three-dimensional coordinate system PM.As used hereinafter, all processes relating to the production ofstereoscopic image-pairs shall be deemed to occur within "3-D Image-PairProduction Space (RA)" indicated in FIGS. 3A, 5A, and 6A; conversely,all processes relating to the viewing of stereoscopic image-pairs shallbe deemed to occur within "3-D Image-Pair Display Space (RB)" alsoindicated in FIGS. 3A, 5A, and 6A. Notably, it is within "3-D Image-PairDisplay Space", that the viewer actually resides and perceivesstereoscopically, with 3-D depth sensation and photo-realism, virtual3-D objects corresponding to either real or synthetic 3-D objects, fromwhich the displayed stereoscopic image-pairs perceived by the viewerhave been produced in 3-D Image-Pair Production Space.

In general, stereoscopically-multiplexed image production subsystem 2may include a stereoscopic-image pair generating (i.e. computing)subsystem 7 illustrated in FIG. 2A, and or a stereoscopic image-pairacquisition subsystem 8 as illustrated in FIG. 2B. Preferably, bothsubsystems are available to stereoscopic image-pair production subsystem2, as this permits the production of stereoscopic image-pairs fromeither real and/or synthetic 3-D objects and scenery, as may be requiredin various present and future VR-applications.

In general, computer models of synthetic objects in 3-D Image ProductionSpace (RA) may be represented using conventional display-list graphicstechniques (i.e. using lists of 3-D geometric equations and parameters)or voxel-based techniques. As illustrated in FIG. 2A, stereoscopicimage-pair generating (i.e. computing) subsystem 7 can producestereoscopic image-pairs from display-list graphics type models and maybe realized using a serial or parallel computing platform of sufficientcomputational performance. In FIG. 2A, one serial computing platform,based on the well know Von Neumann architecture, is show for use inimplementing this subsystem. As shown, subsystem 7 of the illustrativeembodiment comprises a graphics processor 9, information memory 10,arithmetic processor 11, control processor 12 and a system bus 13,arranged as shown and constructed in a manner well known in the art.Commercially available computing systems that may be used to realizethis subsystem include the ONYX, POWER INDIGO2 and CHALLENGE computingsystems from Silicon Graphics, Inc. of Mountain View, Calif.

Alternatively, voxel-based models (m) of synthetic objects may becreated using a parallel computing system of the type disclosed in U.S.Pat. No. 5,361,385 to Bakalash entitled "Parallel Computing System ForVolumetric Modeling, Data Processing and Visualization" incorporatedherein by reference in its entirety. When using voxel-based models ofreal or synthetic 3-D objects, suitable modifications will be made tothe mapping processes mmcl and mmcr generally illustrated in FIG. 3A.

As illustrated in FIG. 2B, the stereoscopic image-pair acquisitionsubsystem 8, of the stereoscopic image-pair production subsystem 2,comprises a number of subcomponents, namely: a miniature opticalplatform 15 with embedded coordinate system pq, for supporting theminiature optical and electro-optical components of subsystem 8; a3-axis platform translation mechanism 16, for rapidly translating theoptical platform 15 with respect to the real object M relative to itsembedded coordinate system pm, in response to translation signalsproduced from platform rotation and position processor 17; left andright perspective image-forming optical assemblies 18 and 19,respectively, for forming left and right wide-angle images 20 and 27 ofa real object M situated in coordinate system pm, imaged from left andright viewing perspectives specified by viewing mappings mmcl and mmcr;left and right image detection/quantization/sampling panels 21 and 23,for detecting, quantizing and spatially sampling the left and rightwide-angle perspective images formed thereon by wide angle opticalassemblies 18 and 19, respectively, and thereby producing a quantizedimage representation of the left and right perspective images as ifviewed by the image-acquisition subsystem using camera parametersspecified by transformations Twd, Tdv, Tvel, Tver; left and right imagescanner/processors 24 and 25, respectively, for scanning and processingthe left and right quantized image representations produced by left andright image detection/quantization/sampling panels 21 and 23,respectively, so as to produce stereoscopic image-pairs {Ipl, Ipr}, on areal-time basis. Notably, the function of stereoscopic image-acquisitionsubsystem 8 is to produce stereoscopic image-pairs viewed using a set ofcamera parameters that correspond to the viewing parameters of thebinocular vision system of the human being viewing the display surfaceof the system. The structure and function of this subsystem will bedescribed in greater detail below.

As shown in FIG. 2B the object, M, is optically imaged on the rightquantization sampler surface, scr, through the right wide angle imagingoptics. This optical imaging is represented by the mapping mmcr. Theright quantization sampler 23 converts the optical image formed on thesurface scr into a pixelized image representation using a quantizationmapping mcpr. The right quantization sampler 23 can be implemented withCCD (charge coupled device) sensor technology which is well known. Thequantization mapping, mcpr, determines which portion of the opticalimage falling on surface scr, shown as image Icr (with imbeddedreference frame pcr), to quantize and is defined by the right and leftquantization mapping processor. The right pixel/scanner processor 25scans the image Icr to produce the pixelized image Ipr.

In a similar manner, the object, M, is optically imaged on the leftquantization sampler 21 surface, scl, through the left wide-angleimaging optics 18. This optical imaging is represented by the mappingmmcl. The left quantization sampler 21 converts an optical image formedon the surface scl into a pixelized image representation using aquantization mapping mcpl. The left quantization sampler can beimplemented with CCD (charge coupled device) sensor technology which iswell known. The quantization mapping, mcpl, determines which portion ofthe optical image falling on surface scl, shown as image Icl (withimbedded reference frame pcl), to quantize and is defined by the rightand left quantization mapping processor 25. The left pixel/scannerprocessor 24 scans the image Icl to produce pixelized image Ipl.

The right and left quantization mapping processors 24 and 25 receive asinput, the eye/head/display tracking information (i.e. transformationsTwd, Tdv, Tvel, and Tver), which are used to define the right and leftquantization mappings, mcpr and mcpl, respectively. Not only do thesetwo quantization mappings define the quantization of the images Icr andIcl, respectively, they also define the size and location of the imagesIcr and Icl, respectively, on the surfaces scr and scl, respectively.The size and location of the images Icr and Icl, within the surfaces scrand scl, defines which portions of the physical object, M, imaged on thequantization samplers are represented by the output images Ipr and Ipl.This flexibility of choosing the size and location of the images Icr andIcl allows each channel, right and left, of the stereoscopicimage-acquisition subsystem 8 to independently "look at" or acquiredifferent portions, along a viewing direction, of the imaged object Mwith the above-described eye/head/display tracking information. In theillustrated embodiment, the mappings mcpr and mcpl, the right and leftpixel scanner/processors 24 and 25, and the right and left quantizationmapping processors 21 and 23 are each implemented by non-mechanicalmeans, thus making it possible to change the "looking" direction ofStereoscopic image acquisition subsystem speeds comparable or betterthan that of the human visual system.

The rotation and position processor 17 controls (i) the three axis"rotation" mechanism 16 (which is responsible for aiming the wide angleoptics and the quantization samplers), and (ii) the 3 axis "translation"mechanism (which is responsible for moving the wide angle optics and thequantization samplers to different locations in the image acquisitionspace RA).

As illustrated in FIG. 2C, eye/head position and orientation trackingsystem 3 comprises a pair of miniature eye/head imaging cameras 30 and31, and a eye/head position and orientation computing subsystem 32. Thefunction of this system is to capture images of the viewers's head 33and eyes 34 and 35, process the same in order to generate, on areal-time basis, head and eye position and orientation parameters foruse in parameterizing the head/eye viewing transformation, Twd, requiredby stereoscopic image-pair production subsystem 8, as illustrated in thegeneral and specific illustrative embodiments of FIGS. 1, 3A and 5A and6A, respectively. As shown in FIG. 1, the left and right eyes of theviewer has an embedded coordinate reference system, indicated by pel andper, respectively, whereas the viewer's head has an embedded coordinatesystem pv, for referencing position and orientation parameters of theviewer's eyes and head with respect to such coordinate systems.

In the illustrative embodiment shown in FIG. 1, eye/head imaging cameras30 and 31 are realized using a pair of infra-red cameras mounted upon anLCD display panel 37, which forms part of the stereoscopic image displaysubsystem 6. Notably, coordinate reference system ptl and ptr areembedded within the image detection surfaces of imaging camera 30 and31, respectively. The eye/head position and orientation computingsubsystem 32, preferably realized using microelectronic technology, ismounted to the display panel, as well. Head and eye position andorientation parameters, required for parameterizing the left eye viewingtransform Tvel, right eye viewing transform Tver and head-displaytransform Tdv, are computed and subsequently transmitted to stereoscopicimage production subsystem 2 preferably by way of electro-magneticposition/orientation signals, which are received at subsystem 2 using abase transceiver 36 well known in the art. For greater details regardingdisplay position and orientation tracking subsystem 3, reference shouldbe made to U.S. Pat. No.5,305,012 issued Apr. 14, 1994 to Sadeg M.Faris, incorporated herein by reference, and to the prior art referencescited therein relating to eye and head tracking technology commerciallyavailable from particular vendors.

As illustrated in FIG. 2D, stereoscopic image display subsystem 6comprises a number of subcomponents, namely: a video display device 37,preferably realized as a planar high-resolution active matrix liquidcrystal display (LCD) panel, capable of displaying color images at ahigh video rate; a video graphics processor 38 for convertingstereoscopically-mulitplexed images (e.g. SMIs) into video signalsadapted to the LCD panel 37; and display driver circuitry 39 for drivingthe pixels on the LCD panel 37 in response to the video signals producedfrom the video processor 38. In the illustrative embodiment of thepresent invention, shown in FIGS. 1, 5A-6B, a micropolarization panel 40is directly mounted onto the planar display surface 41 of LCD panel 37in order to impart polarization state P1 to pixels associated with theleft perspective image in the produced stereoscopic image-pair, and toimpart polarization state P2 to pixels associated with the rightperspective image in the produced stereoscopic image-pair. The structureand function of micropolarization panel 40 is disclosed in copendingapplication Ser. No. 08/126,077, supra, and may be made usingmanufacturing techniques disclosed in U.S. Pat. No. 5,327,285 granted toSadeg M. Faris, and incorporated herein by reference in its entirety.Micropolarization panels of various sizes and spatial resolutions arecommercially available from Reveo, Inc. of Hawthorne, N.Y., under thetrademark μPol™, and can be used to construct the stereoscopic displaypanel of the system of the present invention.

As illustrated in FIG. 2E, display position and orientation trackingsystem 4 comprises a number of spatially distributed components, namely:a display position and orientation signal transmitter 43 remotelylocated from display panel surface 41 and having embedded thereincoordinate reference system px; a display position and orientationsignal receiver 44 mounted on the display panel 37 and having embeddedtherein coordinate reference system pd; and a display position andorientation computing subsystem 45, realized using microelectronictechnology, and mounted to the display panel as shown. The function ofthis system is to sense the position and orientation of the displaysurface 40 with respect to coordinate system px and to generate, on areal-time basis, head and eye position and orientation parameters,transformed to coordinate system px using homogeneous transformationswell known in the art. In turn, these parameters are used inparameterizing the display viewing transformation, Twd, required bystereoscopic image-pair production subsystem 2, as illustrated in thegeneral and specific illustrative embodiments of FIGS. 1, 3A and 5A and6A, respectively.

The display position and orientation parameters, required for displayviewing transform Twd, are transmitted to stereoscopic image productionsubsystem 2 preferably by way of electro-magnetic position/orientationsignals, which are received at subsystem 2 using base transceiver 46well known in the art.

As illustrated in FIG. 2, the stereoscopic image-multiplexing subsystem5 hereof comprises a number of subcomponents, namely: shared memorysubsystem 50 providing various memory storage structures including aKernel and Transform storage unit 51, a left-respective image buffer 52,a right-perspective image buffer 53, and a multiplexed-image buffer 54;a memory access system 55 for accessing (i.e. storing and retrieving)information in various memory structures realized in shared memorysubsystem 50; an image-pair input subsystem 56 for inputting the leftand right perspective images of stereoscopic image-pairs, through memoryaccess subsystem 55, to left and right perspective buffers 52 and 53,respectively; a kernel and transform generation processor 57 forgenerating kernel and transformed utilized through the system andprocess of the present invention; a plurality of N Kernel/Transformprocessors 58A to 58Z for processing pixelated data sets withinquantized image respresenations of left and right perspective images,during the stereoscopic multiplexing processes of the present invention,to produce stereoscopically multiplexed images for display; and amultiplexed-image output subsystem 59 for outputting multiplexed-images(e.g. SMIs) from multiplexed-image buffer 54, to the stereoscopic imagedisplay subsystem 6 of the system.

Having described the structure and function of thestereoscopically-multiplexed image production and display system of thepresent invention, it is appropriate at this juncture to now describethe generalized and particular processes of the present invention which,when carried out on the above-described system in a real-time manner,support realistic stereoscopic viewing of real and/or synthetic 3-Dobjects by viewers who "visually interact" with the stereoscopic imagedisplay subsystem 6 thereof.

In FIG. 3a, the five major sub-systems of a generalized embodiment ofthe system hereof is shown schematically, the object representationsubsystem, the stereoscopic image-pair production subsystem, thestereoscopic image multiplexing subsystem, the stereoscopic image-pairdisplay subsystem, and the stereoscopic image-pair viewing subsystem.

The object representation subsystem comprises means for dynamicallychanging of static object or objects, M, which can be either realphysical objects, Mr, or synthetic objects, Ms. These objects, M, arereferenced to the coordinate frame pm in the image acquisition space RA.

As illustrated in FIG. 3A, the stereoscopic image-pair productionsubsystem comprises the right and left object surfaces scr and scl (withimbedded coordinate frames pcr and pcl), the right and left pixelsurfaces spr and spl, the right and left object mappings mmcr and mmcl,the right and left quanitization mappings mcpr and mcpl, the coordinateframe pq, and the supplemental right and left pixel information dr anddl. Coordinate frame pq is referenced with respect to the coordinateframe pm by the transformation Tmq.

The right object mapping, mmcr, creates an image representation, Icr, ofM, onto the surface scr. In a similar manner, the left object mapping,mmcl, creates an image representation, Icl, of M, onto the surface scl.Both surfaces scr and scl are referenced with respect to coordinateframe pq by transformations Tqcr and Tqcl, respectively. The objectmappings mmcr and mmcl can be either physical optical imaging mappingsor virtual geometric mappings implemented with software or hardware.Images Icr and Icl (on surfaces scr and scl, respectively) can berepresented by physical optical images or geometric representations(hardware or software). The images Icr and Icl taken together form thebeginnings of a stereoscopic image-pair which represent a portion of theobject(s) M. The transforms Tmq, Tqcr, and Tqcl and the mappings mmcrand mmcl are defined such that the resulting images Icr and Icl willlead to the creation of a realistic stereoscopic image-pair in latersteps of this process.

The right quantization mapping, mcpr, describes the conversion of theobject image Icr, into a pixelized image Ipr, on the synthetic surface,spr. The image Ipr can be modified with the supplemental pixelinformation dr. In a similar manner, the left quantization mapping,mcpl, describes the conversion of the object image Icl, into a pixelizedimage Ipl, on the synthetic surface, spl. The image Ipl can be modifiedwith the supplemental pixel information dl. The pixelized images Ipr andIpl are represented by a 2-D array, where each element in the arrayrepresents a spatial pixel in the image and contains spectral data aboutthe pixel. The quantization mappings, mcpr and mcpl, basically indicatehow an arbitrary region of Icr and Icl (respectively) are mapped intothe pixelized images Ipr and Ipl (as shown in FIG. 4a for a black andwhite spectral case).

As illustrated in FIG. 3A, the stereoscopic image multiplexing subsystemcomprises the right and left multiplexing mappings (temporal, spatial,or spectral) mpsr and mpsl, the stereoscopic image surface ss, the rightmultiplexing image Isr, and left multiplexed image Isl, and thecomposite multiplexed image Is. The right multiplexing mapping mpsrdefines the mapping of pixels in Ipr to pixels in Isr. Similarly, theleft multiplexing mapping mpsl defines the mapping of pixels in Ipl topixels in Isl. The images Isr and Isl represent the right and left eyestereoscopic perspectives of the object(s) M. Isr and Isl are formed bythe mappings mpsr and mpsl in such a manor as to be compatible with thestereoscopic image-pair display subsystem. Is is formed from Isr and Islas will be described later.

As illustrated in FIG. 3A, the stereoscopic image-pair display subsystemis comprised of the mapping msd, the display surface sd, the rightstereoscopic display image Idr, the left stereoscopic display image Idl,and the composite stereoscopic display image Id, and the coordinateframe pd. The mapping msd defines the mappings of the pixels of Is ontothe display pixels as represented by Id. The mapping msd can representan optical projection subsystem, can be a one-to-one mapping, or caninclude some scaling factors between the image acquisition space RA andthe image display space RB. The images Idr and Idl form a realisticstereoscopic display image-pair which, when viewed by the stereoscopicimage-pair viewing subsystem, form a realistic representation, M', ofthe object(s) M. The virtual object M', is represented in the imagedisplay space, RB, which is referenced to coordinate frame pw. Thedisplay surface sd had imbedded coordinates pd which are referenced tothe image display space coordinates pw by the transformation Twd.

As illustrated in FIG. 3A, the stereoscopic image-pair viewing subsystemis comprised of the right and left optical imaging mappings mder andmdel, the right viewing surface ser with imbedded coordinate system per,the left viewing surface sel with imbedded coordinate system pel, theright and left viewed images Ier and Iel, the viewing coordinate systempv, and the visual processing subsystem B. The right viewing image Ieris formed on the right viewing surface ser by the right optical imagingmapping mder. The left viewing image Iel is formed on the left viewingsurface sel by the left optical imaging mapping mdel. The relationshipbetween the right an left viewing surfaces and the viewing coordinatesystem, pv, is given by the transformations Tver and Tvel respectively.The relationship between the viewing coordinate system, pv, and thedisplay surface coordinate system pd is given by the transformation Tdv.The transformations Tdv, Tver, and Tvel describe the position andorientation of the right and left viewing surfaces with respect to thedisplay surface sd.

FIG. 3B shows the process steps to be carried out when computing thetransformations and mappings to implement the system shown in FIG. 3A.As shown in FIG. 3B, the process comprises five process groups, labeledA through E, as identified as follows: the object representation processsteps (A), the stereoscopic image-pair generation process steps (B), thestereoscopic image multiplexing process steps (C), the stereoscopicimage-pair display process steps (D), and the stereoscopic image-pairviewing process steps (E).

The object representation process operates on an object representationof either a real physical object(s), Mr, or synthetic objectrepresentations, Ms. The synthetic object representations, Ms, can berepresented in any convenient form such as the common geometricrepresentations used in polygonal modeling systems (vertices, faces,edges, surfaces, and textures) or parametric function representationsused in solids modeling systems (set of equations) which is well knownin the art. The result of the object representation process steps is thecreation of an object representation M which is further processed by thestereoscopic image-pair production process steps.

The stereoscopic image-pair generation process steps operate on theobject representation, M and produces the right and left pixelizedstereoscopic image-pairs Ipr and Ipl, respectively. The steps of thestereoscopic image-pair generation process use the transformations Tdv(acquired by the head position and orientation tracking subsystem), Tveland Tver (acquired by the eye position and orientation trackingsubsystem), and Twd (acquired by the display position and orientationtracking subsystem) and the acquisition of the display parameters msd,ss, and sd to compute various transformations and mappings as will bedescribed next.

The transformation Tmq describes the position and orientation placementof the right and left stereoscopic image-pair acquisition surfaces scrand scl. Tmq is computed by the function fTmq which accepts asparameters Twd, Tdv, Tvel, Tver, and pm. fTmq computes Tmq such that theimages Icr and Icl taken together form the beginnings of a stereoscopicimage-pair which represents a portion of the object(s) M.

The transformation Tqcr describes the position and orientation placementof the right stereoscopic image-pair acquisition surface Icr withrespect to pq. Tqcr is computed by the function fTqcr which accepts asparameters Tmq, Tdv, Tvel, and Tver. fTqcr computes Tqcr such that theimage Icr from the surface scr forms the beginnings of a realisticstereoscopic image-pair. In a similar manor, the transformation Tqcldescribes the position and orientation placement of the leftstereoscopic image-pair acquisition surface Icl with respect to pq. Tqclis computed by the function fTqcl which accepts as parameters Tmq, Tdv,Tvel, and Tver. fTqcl computes Tqcl such that the image Icl from thesurface scl forms the beginnings of a realistic stereoscopic image-pair.

The right object mapping, mmcr, creates an image representation, Icr, ofM, onto the surface scr. mmcr is computed by the function fmmcr whichaccepts as parameters Tmq, Tqcr, Tdv, Tver, sd, and msd. mmcr representseither a real optical imaging process in the case of a real object Mr ora synthetic rendering process (well known in the art) in the case of asynthetic object Ms. In a similar manor, the left object mapping, mmcl,creates an image representation, Icl, of M, onto the surface scl. mmclis computed by the function fmmcl which accepts as parameters Tmq, Tqcl,Tdv, Tvel, sd, and msd. mmcl represents either a real optical imagingprocess in the case of a real object Mr or a synthetic rendering process(well known in the art) in the case of a synthetic object Ms.

The image representation Icr on surface scr is formed by the functionfIcr which accepts as parameters mmcr and M. In a similar manor, theimage representation Icl on surface scl is formed by the function fIclwhich accepts as parameters mmcl and M. Mappings mmcr and mmcl aredefined in such a way that the images Icr and Icl taken together formthe beginnings of a realistic right and left, respectively, stereoscopicimage-pair which represents a portion of the object(s) M.

The right quantization mapping, mcpr, describes the conversion of theobject image Icr, into a pixelized image Ipr, on the synthetic surface,spr. mcpr is computed by the function fmcpr which accepts as parametersTdv, Tvel, Tver, sd, ss, and msd. The quantization mapping, mcprindicates how an arbitrary region of Icr is mapped into the pixelizedimage Ipr. In a similar manor, the left quantization mapping, mcpl,describes the conversion of the object image Icl, into a pixelized imageIpl, on the synthetic surface, spl. mcpl is computed by the functionfmcpl which accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd. Thequantization mapping, mcpl indicates how an arbitrary region of Icl ismapped into the pixelized image Ipl.

The right pixelized image Ipr on surface spr is formed by the functionfIpr which accepts as parameters Icr, mcpr, and dr, where dr representssupplemental pixel information. In a similar manor, the left pixelizedimage Ipl on surface spl is formed by the function fIpl which accepts asparameters Icl, mcpl, and dl, where dl represents supplemental pixelinformation. Mappings mcpr and mcpl are defined in such a way that theresulting images Ipr and Ipl will lead to the creation of a realisticstereoscopic image-pair in later steps of this process. Mappings mcprand mcpl can also be used to correct for limitations of an implementedsystem for performing the mappings mmcr and mmcl described above.

The stereoscopic image multiplexing process steps operate on the rightand left pixelized images Ipr and Ipl respectively and produces theright and left multiplexed stereoscopic image representations Isl andIsr. The steps of the stereoscopic image-pair multiplexing process usethe transformations Tdv (acquired by the head position and orientationtracking subsystem), and Tvel and Tver (acquired by the eye position andorientation tracking subsystem), and the acquisition of the displayparameters msd, ss, and sd to compute various transformations andmappings as will be described next.

The right multiplexing mapping, mpsr, defines the mapping of pixels inIpr to pixels in Isr. mpsr is computed by the function fmpsr whichaccepts as parameters Tdv, Tvel, Tver, sd, ss, and msd. In a similarmanor, the left multiplexing mapping, mpsl, defines the mapping ofpixels in Ipl to pixels in Isl. mpsl is computed by the function fmpslwhich accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd.

The right multiplexed image Isr, on surface ss, is formed by thefunction fIsr which accepts as parameters Ipr and mpsr. Likewise, theleft multiplexed image Isl, on surface ss, is formed by the functionfIsl which accepts as parameters Ipl and mpsl. Isr and Isl are formed bythe mappings mpsr and mpsl in such a manor as to be compatible with thestereoscopic image-pair display subsystem. The composite multiplexedstereoscopic image, Is, is formed from the compositing of Isr and Isl.

The stereoscopic image-pair display process steps operate on the rightand left stereoscopic images Isr and Isl, respectively, using thedisplay mapping msd, to display the right and left stereoscopic imagedisplay pairs Idr and Idl. The mapping msd can be an electronics mappingto pixels on a display or projection optics to image onto a screen.

The right stereoscopic display image Idr, on surface sd, is formed bythe function/process fIdr which accepts as parameters Isr and msd.Likewise, the left stereoscopic display image Idl, on surface sd, isformed by the function/process fIdl which accepts as parameters Isl andmsd. The function/processes fIdr and fIdl form the stereoscopic encodingprocess which encodes the right and left stereoscopic display images,Idr and Idl in a form which can be viewed in a stereoscopic viewing modeby the stereoscopic image-pair viewing process or processes. Thecomposite multiplexed stereoscopic display image, Id, is formed from thecompositing of Idr and Idl.

The stereoscopic display surface, sd, has imbedded coordinates pd whichare related to pw by the transformation Twd. The display position andorientation tracking process tracks the interaction of the display withthe virtual environment M' and acquires the transformation Twd.

The stereoscopic image-pair viewing process steps represent the viewingdecoding of the right and left stereoscopic display images, Idr and Idl,through the decoding mappings mder and mdel, respectively, to producethe right and left viewer images, Ier and Iel, respectively. The rightand left viewer Images Ier and Iel, are formed on the right and leftviewing surfaces ser and sel, respectively. The right viewing surface,ser, has imbedded coordinate frame per. Coordinate frame per is relatedto frame pv by the transformation Tver. Likewise, the left viewingsurface, sel, has imbedded coordinate frame pel. Coordinate frame pel isrelated to frame pv by the transformation Tvel. The function/processfIpr accepts parameters Idr, and mder and performs the actual decodingof the image Idr to form the image Ier. Likewise, the function/processfIpl accepts parameters Idl, and mdel and performs the actual decodingof the image Idl to form the image Iel. The combination of images Ierand Iel in the visual processing center, B, forms the image Ib. Ibrepresents the perceived stereoscopic image M' as represented in thevisual processing center B through the use of the function/process fIb.

Coordinate frame pv represents the imbedded coordinate frame of thecombined right and left viewing surfaces ser and sel, respectively. pvis related to the stereoscopic image-pair display coordinates system,pd, by the transformation Tdv.

The head position and orientation tracking process tracks theinteraction of the combined right and left viewing surfaces, ser andsel, with the display surface, sd, and acquires the transformation Tdvto describe this interaction. The eye position and orientation trackingprocess tracks the interaction of each individual right and left viewingsurface, ser and sel, with respect to the coordinate frame pv, andacquires the right and left viewing surface transformations, Tver andTvel.

The overall process steps set forth in the process groups A through E inFIG. 3B defines an interactive process which starts with the acquisitionof the transformations Tdv (acquired by the head position andorientation tracking subsystem), Tvel and Tver (acquired by the eyeposition and orientation tracking subsystem), and Twd (acquired by thedisplay position and orientation tracking subsystem) and the acquisitionof the display parameters msd, ss, and sd. The interactive processcontinues with the process steps A, B, C, D, and E in the given orderand then repeats with the acquisition of the above transformations andparameters.

Referring to FIGS. 2A, 4, 4B, 4C, the subsystem and generalizedprocesses for producing stereoscopically-multiplexed images fromstereoscopic image-pairs, will now be described.

FIG. 4B shows the pixelized image representation for images Ipl, Ipr,Isl, Isr, Is, Idl, Idr, and Id. By definition, an image is a collectionof (K horizontal by L vertical) pixels where each pixel block representsa spectral vector with N or P elements (N for images Ipl and Ipr and Pfor Isl and Isr). The elements of the spectral vector represent thespectral characteristics of the pixel in question (color, intensity,etc.). A typical representation would be a 3×1 vector representing thered, blue, and green components of the pixel. The pixels are indexedusing an X,Y coordinate frame as shown in the figure. FIG. 4C shows showhow the spatial kernel is applied to an image. A spatial kerneltransforms a group of pixels in an input image into a single pixel in anoutput image. Each pixel in the output image (x,y) at a particular time(t) has a kernel K(x,y,t) associated with it. This kernel defines theweights of the neighboring pixels to combine to form the output imageand is an N×N matrix where N is odd. The center element in the kernelmatrix is the place holder and is used to define the relative positionof the neighboring pixels based on the place holder pixel whichrepresents x,y. In the example in FIG. 4C, element e is the placeholder. Each element in the kernel matrix indicates a scaling weight forthe corresponding pixel (this weight is applied to the spectral vectorin that pixel location, v(x,y)). The pairing between the scaling weightsand the pixel values is determined by the relative location to the placeholder pixel. For example, the 3×3 kernel above is associated with theoutput pixel (7,3) so the element `e` in the matrix defines the weightof the input pixel (7,3) and the neighboring elements in the kerneldefines the 8 neighboring pixel weights based on their location in thematrix. If the spectral vectors are represented by vi(x,y,t) for theinput image and vo(x,y,t) for the output image, then the output spectralvector vo(7,3,t) would be defined as follows (dropping the timedependence for ease of notation):

    vio(7,3)=avi(6,2)+bvi(7,2)+cvi(8,2)+dvi(6,3)+evi(7,3)+

    fvi(8,3)+gvi(6,4)+hvi(7,4)+ivi(8,4)

The quantized images, Ipl and Ipr, can be represented by a 2-D array ofpixels, Ipl(x,y,t) and Ipr(x,y,t), where each x,y pixel entry representsa quantized area of the images Icl and Icr at quantized time t. The timevalue, t, in the Ipl and Ipr images, represents the time component ofthe image. The time value, image time, increments by 1 each time the Ipland Ipr images change. Each image pixel is represented by a spectralvector, v(x,y,t), of size N×1 or P×1 (N for images Ipl and Ipr and P forIsl and Isr). A typical representation would be a 3×1 vectorrepresenting the red, blue, and green components of the pixel.Individual values in the intensity vector are represented as v(x,y,i)where i is the particular intensity element.

In essence, the general stereoscopically-multiplexed image process mapsthe pixels of Ipl into the pixels of Isl using the mapping mpsl andmapping the pixels of Ipr into the pixels of Isr using the mapping mpsr.The images Isl and Isr can be combined into the composite image Is orcan be left separate. The general stereoscopically-multiplexed imageprocess can be described by the following equations:

    Isl(x,y,t)=Rl(x,y,t,Ipl(t))Kl(x,y,t){Ipl(t)}               1.

    Isr(x,y,t)=Rr(x,y,t,Ipr(t))Kr(x,y,t){Ipr(t)}               2.

    Is(x,y,t)=cl(t)Isl(x,y,t)+cr(t)Isr(x,y,t)

where

Isl(t) is the entire left output image from the multiplex operation attime t,

Isr(t) is the entire right output image from the multiplex operation attime t,

Rl is the spectral transform matrix which is a function of the pixelspectral vector in question (x,y) at time t. Rl is also a function ofIpl(t) which allows the transform to modify itself based on the spectralcharacteristics of the pixel or neighboring pixels. Typically onlyIpl(x,y,t) would be used and not Ipl(t).

Rr is the spectral transform matrix which is a function of the pixelspectral vector in question (x,y) at time t. Rr is also a function ofIpr(t) which allows the transform to modify itself based on the spectralcharacteristics of the pixel or neighboring pixels. Typically onlyIpr(x,y,t) would be used and not Ipr(t).

Kl is the spectral kernel operating on Ipl(t). Kl is a function of thecurrent position in question (x,y) and the current time t.

Kr is the spectral kernel operating on Ipr(t). Kr is a function of thecurrent position in question (x,y) and the current time t.

Ipl(t) is the entire Ipl image, Ipr(t) is the entire Ipr image.

Isl and Isr are the left and right multiplex output images,respectively.

Is is the composite of the two image Isl and Isr. This step is optional,some stereoscopic display systems can use a single stereoscopic imagechannel, Is, and others require separate left and right channels, Island Isr.

The operations represented by Equations 1 and 2 above are evaluated foreach x,y pixel in Isl and Isr (this process can be performed in parallelfor each pixel or for groups of pixels). The operations are performed intwo steps. First, the spatial kernel Kl(x,y,t) is applied to the imageIpl(t) which forms a linear combination of the neighboring pixels ofIpl(x,y,t) to produce a spectral vector vl(x,y,t) at (x,y) at time t.Second, this spectral vector vl(x,y,t) is multiplied by the spectraltransformation matrix, Rl(x,y,t,Ipl), to produce a modified spectralvector which is stored in Isl(x,y,t). This process is carried out foreach pixel, (x,y), in Isl(t). The same process is carried out for eachpixel in Isr(x,y,t). The resulting images, Isl(t) and Isr(t) can becomposited into a single channel Is(t) by equation 3 above by a simplelinear combination using weights cl(t)and cr(t)which are functions oftime t. Typically, cl(t)=cr(t)=1. Advantageously, Equations 1, 2, and 3can be evaluated in a massively parallel manner.

The spatial kernels, Kl and Kr, are N×N matrices where N is odd, andeach entry in the matrix represents a linear weighting factor used tocompute a new pixel value based on its neighbors (any number of them). Aspatial kernel transforms a group of pixels in an input image into asingle pixel in an output image. Each pixel in the output image (x,y) ata particular time (t) has a kernel K(x,y,t) associated with it. Thiskernel defines the weights of the neighboring pixels to combine to formthe output image and is an N×N matrix where N is odd. The center elementin the kernel matrix is the place holder and is used to define therelative position of the neighboring pixels based on the place holderpixel which represents x,y.

Notably, using massively parallel computers and a real-timeelectronically adapting micorpolarization panel 41, it is possible toachieve an adaptive encoding system which changes the micropolarizationpattern (e.g. P1, P2, P1, P2, etc.) upon display surface 40, as requiredby the symmetries of the image at time (t).

Below are three examples of possible kernel functions that may be usedwith the system of the present invention to produce and display (1)spatially-multiplexed images (SMI) using a 1-D spatial modulationfunction, (2) temporally-multiplexed images, and (3) a SMI using a 2-Dspatial modulation function. Each of these examples will be consideredin their respective order below.

In the first SMI example, the kernel has the form: ##EQU1##

This happens to be the left image Kernel for the 1-D spatialmultiplexing format. Note that the above kernel is not a function oftime and it therefor a spatial multiplexing technique. The center entrydefines the pixel in question and the surrounding entries define theweights of the neighboring pixels to combine. Each corresponding pixelvalue is multiplied by the weighting value and the collection is summed.The result is the spectral vector for the Is image. In the above case,we are averaging the current pixel with the pixel above it on odd linesand doing nothing for even lines. The corresponding Kr for the SMIformat is given below: ##EQU2##

In the second example, the kernel functions for the field sequential(temporal multiplexing) technique are provided by the followingexpressions:

    Kl(x,y,t)=[1] for frac(t)<0.5

    Kl(x,y,t)=[0] for frac(t)>0.5

    Kr(x,y,t)=[1] for frac(t)>0.5

    Kr(x,y,t)=[0] for frac(t)<0.5

where, frac(t) returns the fractional portion of the time value t. Theseexpressions state that left pixels are only "seen" for the first half ofthe time interval frac(t) and right pixels for the second half of thetime interval frac(t).

The third example might be used when a pixel may be mapped into morethan one pixel in the output image, as in a 2-D a checker boardmicropolarization pattern. An example kernel function, Kr, for acheckerboard polarization pattern might look like the following:##EQU3##

Having described these types of possible kernels that may be used in thestereoscopic multiplexing process, attention is now turned to thespectral transformation matrices, Rl and Rr, which addresses thestereoscopic-multiplexing of the spectral components of left and rightquantized perspective images.

The spectral transformation matrices, Rl and Rr, define a mapping of thespectral vectors produced by the kernel operation above to the spectralvectors in the output images, Isl and Isr. The spectral vectorrepresentation used by the input images, Ipl and Ipr, do not need tomatch the spectral vector representation used by the output images, Island Isr. For example, Ipl and Ipr could be rendered in full color andIsl and Isr could be generated in gray scale. The elements in thespectral vector could also be quantized to discrete levels. The spectraltransformation matrices are a function of the x,y pixel in question attime t and also of the entire input image Ip. The parameterization of Rland Rr on Ipl and Ipr (respectively) allows the spectral transformationto be a function of the color of the pixel (and optionally neighboringpixels) in question. By definition, a spectral transformation matrix isa P×N matrix where N is the number of elements in the spectral vectorsof the input image and P is the number of elements in the spectralvectors of the output image. For example, if the input image had a 3×1red, green, blue spectral vector and the output image was gray scale,1×1 spectral vector, a spectral transform matrix which would convert theinput image into a b/w image might look like, Rl(x,y,t)=[0.3 0.45 0.25]which forms the gray scale pixel by summing 0.3 times the red component,0.45 times the green component, and 0.25 times the blue component. Acolor multiplexing system with a red, green, and blue 3×1 spectralvector might look like this:

    Rl(x,y,t)=diag(1 0 1) for frac(t)<0.5

    Rl(x,y,t)=diag(0 1 0) for frac(t)>0.5

    Rr(x,y,t)=diag(0 1 0) for frac(t)<0.5

    Rr(x,y,t)=diag(1 0 1) for frac(t)>0.5

Where diag(a,b,c) is the 3×3 diagonal matrix with diagonal elements a,b, c. A spectral transformation matrix to create a red/green anaglyphstereoscopic image could be: ##EQU4##

Note, in the above cases, when the spectral kernels are not specifiedthey are assumed to be Kl=[1] and Kr=[1].

In FIG. 5A, there is shown another embodiment of the system hereof,which uses a novel spatial multiplexing technique and stereoscopic imagepair generation system 7 viewing synthetic objects, Ms. As shown, thesystem is divided into five sub-systems, namely the objectrepresentation subsystem, the stereoscopic image-pair generationsubsystem, the stereoscopic image multiplexing subsystem (using spatialmultiplexing processes), the stereoscopic image-pair display subsystembased on the micro-polarizer technology, and the stereoscopic image-pairviewing subsystem based on polarization decoding techniques.

The object representation subsystem is comprises of dynamically changingor static object or objects, M, which are synthetic objects, Ms. Theseobjects, M, are referenced to the coordinate frame pm in the imageacquisition space RA.

The stereoscopic image-pair production subsystem is comprised of theright and left object surfaces scr and scl (with imbedded coordinateframes pcr and pcl), the right and left pixel surfaces spr and spl, theright and left object mappings mmcr and mmcl, the right and leftquanitization mappings mcpr and mcpl, the coordinate frame pq, and thesupplemental right and left pixel information dr and dl. Coordinateframe pq is referenced with respect to the coordinate frame pm by thetransformation Tmq.

The right object mapping, mmcr, creates an image representation, Icr, ofM, onto the surface scr. In a similar manner, the left object mapping,mmcl, creates an image representation, Icl, of M, onto the surface scl.Both surfaces scr and scl are referenced with respect to coordinateframe pq by transformations Tqcr and Tqcl, respectively. The objectmappings mmcr and mmcl are virtual geometric mappings implemented withsoftware or hardware. Images Icr and Icl (on surfaces scr and scl,respectively) are represented by geometric representations (hardware orsoftware). The images Icr and Icl taken together form the beginnings ofa stereoscopic image-pair which represent a portion of the virtualobject(s) M. The transforms Tmq, Tqcr, and Tqcl and the mappings mmcrand mmcl are defined such that the resulting images Icr and Icl willlead to the creation of a realistic stereoscopic image-pair in latersteps of this process.

The right quantization mapping, mcpr, describes the conversion of theobject image Icr, into a pixelized image Ipr, on the synthetic surface,spr. The image Ipr can be modified with the supplemental pixelinformation dr. In a similar manner, the left quantization mapping,mcpl, describes the conversion of the object image Icl, into a pixelizedimage Ipl, on the synthetic surface, spl. The image Ipl can be modifiedwith the supplemental pixel information dl. The pixelized images Ipr andIpl are represented by a 2-D array, where each element in the arrayrepresents a spatial pixel in the image and contains spectral data aboutthe pixel. The quantization mappings, mcpr and mcpl, indicate how anarbitrary region of Icr and Icl (respectively) are mapped into thepixelized images Ipr and Ipl.

The stereoscopic image multiplexing subsystem supports the right andleft multiplexing mappings mpsr and mpsl, the stereoscopic image surfacess, the right multiplexing image Isr, and left multiplexed image Isl,and the composite multiplexed image Is. The right multiplexing mappingmpsr defines the spatial mapping of pixels in Ipr to pixels in Isr.Similarly, the left multiplexing mapping mpsl defines the spatialmapping of pixels in Ipl to pixels in Isl. The images Isr and Islrepresent the right and left eye stereoscopic perspectives of theobject(s) M. Isr and Isl are formed by the mappings mpsr and mpsl insuch a manner as to be compatible with the micro-polarizer basedstereoscopic image-pair display subsystem. Is is formed from Isr and Islas will be described later.

The stereoscopic image-pair display subsystem supports the mapping msd,the display surface sd, the right stereoscopic spatial multiplexeddisplay image Idr, the left stereoscopic spatial multiplexed displayimage Idl, and the composite stereoscopic spatial multiplexed displayimage Id, and the coordinate frame pd. The mapping nsd defines themapping of the pixels of Is onto the display pixels as represented byId. The mapping msd represents an optical projection subsystem and caninclude some scaling factors between the image acquisition space RA andthe image display space RB. The images Idr and Idl form a realisticspatially multiplexed stereoscopic display image-pair which, when viewedby the stereoscopic image-pair viewing subsystem, form a realisticrepresentation, M', of the object(s) M. The virtual object M', isrepresented in the image display space, RB, which is referenced tocoordinate frame pw. The display surface sd, contains a micro-polarizerarray which performs polarization encoding of the images Idr and Idl.Surface sd has imbedded coordinates pd which are referenced to the imagedisplay space coordinates pw by the transformation Twd.

The stereoscopic image-pair viewing subsystem supports the right andleft optical imaging mappings mder and mdel, the right viewing surfaceser with imbedded coordinate system per, the left viewing surface selwith imbedded coordinate system pel, the right and left viewed imagesIer and Iel, the viewing coordinate system pv, and the visual processingsubsystem B. The right viewing image Ier is formed on the right viewingsurface ser by the right optical imaging mapping mder which performs apolarization decoding process. The left viewing image Iel is formed onthe left viewing surface sel by the left optical imaging mapping mdelwhich performs a polarization decoding process. The relationship betweenthe right an left viewing surfaces and the viewing coordinate system,pv, is given by the transformations Tver and Tvel respectively. Therelationship between the viewing coordinate system, pv, and the displaysurface coordinate system pd is given by the transformation Tdv. Thetransformations Tdv, Tver, and Tvel describe the position andorientation of the right and left viewing surfaces with respect to thedisplay surface sd.

FIG. 5B shows the process steps to be carried out when computing thetransformations and mappings to implement the system shown in FIG. 5A.The process steps are organized into five process groups labeled Athrough E in FIG. 5B and are indicated by the object representationprocess steps (A); the stereoscopic image-pair generation process steps(B); the stereoscopic image multiplexing process steps (carried outusing spatial multiplexing) (C); the stereoscopic image-pair display(micro-polarization filter based) process steps (D); and thestereoscopic image-pair viewing process steps (based on polarizationdecoding processes) (E).

The object representation process steps operate on an objectrepresentation of a synthetic object, Ms. The synthetic objectrepresentations, Ms, can be represented in any convenient form such asthe common geometric representations used in polygonal modeling systems(vertices, faces, edges, surfaces, and textures) or parametric functionrepresentations used in solids modeling systems (set of equations) whichis well known in the art. The result of the object representationprocess steps is the creation of an object representation M which isfurther processed by the stereoscopic image-pair generation processsteps.

The stereoscopic image-pair generation process steps operate on theobject representation, M and produces the right and left pixelizedstereoscopic image-pairs Ipr and Ipl, respectively. The steps of thestereoscopic image-pair generation process use the transformations Tdv(acquired by the head position and orientation tracking subsystem), Tveland Tver (acquired by the eye position and orientation trackingsubsystem), and Twd (acquired by the display position and orientationtracking subsystem) and the acquisition of the display parameters msd,ss, and sd to compute various transformations and mappings as will bedescribed next.

The transformation Tmq describes the position and orientation placementof the right and left stereoscopic image-pair generation surfaces scrand scl. Tmq is computed by the function fTmq which accepts asparameters Twd, Tdv, Tvel, Tver, and pm. fTmq computes Tmq such that theimages Icr and Icl taken together form the beginnings of a stereoscopicimage-pair which represents a portion of the object(s) M.

The transformation Tqcr describes the position and orientation placementof the right stereoscopic image-pair acquisition surface Icr withrespect to pq. Tqcr is computed by the function fTqcr which accepts asparameters Tmq, Tdv, Tvel, and Tver. fTqcr computes Tqcr such that theimage Icr from the surface scr forms the beginnings of a realisticstereoscopic image-pair. In a similar manor, the transformation Tqcldescribes the position and orientation placement of the leftstereoscopic image-pair acquisition surface Icl with respect to pq. Tqclis computed by the function fTqcl which accepts as parameters Tmq, Tdv,Tvel, and Tver. fTqcl computes Tqcl such that the image Icl from thesurface scl forms the beginnings of a realistic stereoscopic image-pair.

The right object mapping, mmcr, creates an image representation, Icr, ofM, onto the surface scr. mmcr is computed by the function fmmcr whichaccepts as parameters Tmq, Tqcr, Tdv, Tver, sd, and msd. mmcr representsa synthetic rendering process (well known in the art). In a similarmanner, the left object mapping, mmcl, creates an image representation,Icl, of M, onto the surface scl. mmcl is computed by the function fmmclwhich accepts as parameters Tmq, Tqcl, Tdv, Tvel, sd, and msd. mmclrepresents a geometric rendering process (well known in the art).

The image representation Icr on surface scr is formed by the functionfIcr which accepts as parameters mmcr and M. In a similar manner, theimage representation Icl on surface scl is formed by the function fIclwhich accepts as parameters mmcl and M. Mappings mmcr and mmcl aredefined in such a way that the images Icr and Icl taken together formthe beginnings of a realistic right and left, respectively, stereoscopicimage-pair which represents a portion of the object(s) M.

The right quantization mapping, mcpr, describes the conversion of theobject image Icr, into a pixelized image Ipr, on the synthetic surface,spr. mcpr is computed by the function fmcpr which accepts as parametersTdv, Tvel, Tver, sd, ss, and msd. The quantization mapping, mcprindicates how an arbitrary region of Icr is mapped into the pixelizedimage Ipr. In a similar manner, the left quantization mapping, mcpl,describes the conversion of the object image Icl, into a pixelized imageIpl, on the synthetic surface, spl. mcpl is computed by the functionfmcpl which accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd. Thequantization mapping, mcpl indicates how an arbitrary region of Icl ismapped into the pixelized image Ipr.

The right pixelized image Ipr on surface spr is formed by the functionfIpr which accepts as parameters Icr, mcpr, and dr, where dr representssupplemental pixel information. In a similar manor, the left pixelizedimage Ipl on surface spl is formed by the function fIpl which accepts asparameters Icl, mcpl, and dl, where dl represents supplemental pixelinformation. Mappings mcpr and mcpl are defined in such a way that theresulting images Ipr and Ipl will lead to the creation of a realisticstereoscopic image-pair in later steps of this process. Mappings mcprand mcpl can also be used to correct for limitations of an implementedsystem for performing the mappings mmcr and mmcl described above.

The stereoscopic image spatial multiplexing process steps operate on theright and left pixelized images Ipr and Ipl respectively and producesthe right and left spatially multiplexed stereoscopic imagerepresentations Isr and Isl. The steps of the stereoscopic image-pairspatial multiplexing process use the transformations Tdv (acquired bythe head position and orientation tracking subsystem), and Tvel and Tver(acquired by the eye position and orientation tracking subsystem), andthe acquisition of the display parameters msd, ss, and sd to computevarious transformations and mappings as will be described next.

The right multiplexing mapping, mpsr, defines the mapping of pixels inIpr to pixels in Isr. mpsr is computed by the function fmpsr whichaccepts as parameters Tdv, Tvel, Tver, sd, ss, and msd. In a similarmanner, the left multiplexing mapping, mpsl, defines the mapping ofpixels in Ipl to pixels in Isl. mpsl is computed by the function fmpslwhich accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd.

The right multiplexed image Isr, on surface ss, is formed by thefunction fIsr which accepts as parameters Ipr and mpsr. Likewise, theleft multiplexed image Isl, on surface ss, is formed by the functionfIsl which accepts as parameters Ipl and mpsl. Isr and Isl are formed bythe mappings mpsr and mpsl to be compatible with the micro-polarizingfilter based stereoscopic image-pair display subsystem. The compositemultiplexed stereoscopic image, Is, is formed from the compositing ofIsr and Isl.

The stereoscopic image-pair display process steps operate on the rightand left stereoscopic images Isr and Isl, respectively, using thedisplay mapping msd, to display the right and left stereoscopic imagedisplay pairs Idr and Idl on the micro-polarizer based display surface,sd. The mapping msd represent projection optics.

The right stereoscopic display image Idr, on surface sd, is formed bythe function/process fIdr which accepts as parameters Isr and msd.Likewise, the left stereoscopic display image Idl, on surface sd, isformed by the function/process fIdl which accepts as parameters Isl andmsd. The function/processes fIdr and fIdl form the stereoscopic encodingprocess which encodes the right and left stereoscopic display images,Idr and Idl, using polarized light (via the application of amicro-polarizer to the display surface sd) so at to be viewed in astereoscopic viewing mode by the stereoscopic image-pair viewing processor processes. The composite multiplexed stereoscopic display image, Id,is formed from the compositing of Idr and Idl.

The stereoscopic display surface, sd, has imbedded coordinates pd whichare related to pw by the transformation Twd. The display position andorientation tracking process tracks the interaction of the display withthe virtual environment M' and acquires the transformation Twd.

The stereoscopic image-pair viewing process steps represent the viewingdecoding of the right and left stereoscopic display images, Idr and Idl,through the decoding mappings mder and mdel, respectively, to producethe right and left viewer images, Ier and Iel, respectively. The rightand left viewer Images Ier and Iel are formed on the right and leftviewing surfaces ser and sel, respectively. The right viewing surface,ser, has imbedded coordinate frame per. Coordinate frame per is relatedto frame pv by the transformation Tver. Likewise, the left viewingsurface, sel, has imbedded coordinate frame pel. Coordinate frame pel isrelated to frame pv by the transformation Tvel. The function/processfIpr accepts parameters Idr and mder and performs the actual decoding ofthe image Idr to form the image Ier using a polarizing filter decoder.Likewise, the function/process fIpl accepts parameters Idl and mdel andperforms the actual decoding of the image Idl to form the image Ielusing a polarizing filter decoder. The combination of images Ier and Ielin the visual processing center, B, forms the image Ib. Ib representsthe perceived stereoscopic image M' as represented in the visualprocessing center B through the use of the function/process fIb.

Coordinate frame pv represents the imbedded coordinate frame of thecombined right and left viewing surfaces ser and sel, respectively. pvis related to the stereoscopic image-pair display coordinates system,pd, by the transformation Tdv.

The head position and orientation tracking process tracks theinteraction of the combined right and left viewing surfaces, ser andsel, with the display surface, sd, and acquires the transformation Tdvto describe this interaction. The eye position and orientation trackingprocess tracks the interaction of each individual right and left viewingsurface, ser and sel, with respect to the coordinate frame pv, andacquires the right and left viewing surface transformations, Tver andTvel.

The overall process steps defined by the process groups A through E inFIG. 5B illustrate an interactive process which starts with theacquisition of the transformations Tdv (acquired by the head positionand orientation tracking subsystem), Tvel and Tver (acquired by the eyeposition and orientation tracking subsystem), and Twd (acquired by thedisplay position and orientation tracking subsystem) and the acquisitionof the display parameters msd, ss, and sd. The interactive processcontinues with the process steps A, B, C, D, and E in the given orderand then repeats with the acquisition of the above transformations andparameters.

FIG. 6A shows another embodiment the system of the present inventionspatial-multiplexing technique, and stereoscopic image-pair acquisitionsystem 8 to capture information about real object, Mr. As shown, thesystem involves five sub-systems, namely the object representationsubsystem, the stereoscopic image-pair acquisition subsystem, thestereoscopic multiplexing image subsystem (using spatial-multiplexingprocesses), the stereoscopic image-pair display subsystem 6, and thestereoscopic image-pair viewing subsystem based on polarization decodingtechniques.

In this embodiment, images are dynamically changing or static object orobjects, M, composed of real objects, Mr, are imaged by subsystem 8.These objects, M, are referenced to the coordinate frame pm in the imageacquisition space RA.

The stereoscopic image-pair acquisition subsystem supports the right andleft object surfaces scr and scl (with imbedded coordinate frames pcrand pcl), the right and left pixel surfaces spr and spl, the right andleft object mappings mmcr and mmcl, the right and left quanitizationmappings mcpr and mcpl, the coordinate frame pq, and the supplementalright and left pixel information dr and dl. Coordinate frame pq isreferenced with respect to the coordinate frame pm by the transformationTmq.

The right object mapping, mmcr, creates an image representation, Icr, ofM, onto the surface scr. In a similar manner, the left object mapping,mmcl, creates an image representation, Icl, of M, onto the surface scl.Both surfaces scr and scl are referenced with respect to coordinateframe pq by transformations Tqcr and Tqcl, respectively. The objectmappings mmcr and mmcl are optical imaging processes. Images Icr and Icl(on surfaces scr and scl, respectively) are represented by physicaloptical images. The images Icr and Icl taken together form thebeginnings of a stereoscopic image-pair which represent a portion of thevirtual object(s) M. The transforms Tmq, Tqcr, and Tqcl and the mappingsmmcr and mmcl are defined such that the resulting images Icr and Iclwill lead to the creation of a realistic stereoscopic image-pair inlater steps of this process.

The right quantization mapping, mcpr, describes the conversion of theobject image Icr, into a pixelized image Ipr, on the synthetic surface,spr. The image Ipr can be modified with the supplemental pixelinformation dr. In a similar manner, the left quantization mapping,mcpl, describes the conversion of the object image Icl, into a pixelizedimage Ipl, on the synthetic surface, spl. The image Ipl can be modifiedwith the supplemental pixel information dl. The pixelized images Ipr andIpl are represented by a 2-D array, where each element in the arrayrepresents a spatial pixel in the image and contains spectral data aboutthe pixel. The quantization mappings, mcpr and mcpl, indicate how anarbitrary region of Icr and Icl (respectively) are mapped into thepixelized images Ipr and Ipl.

The stereoscopic image multiplexing subsystem supports the right andleft multiplexing mappings mpsr and mpsl, the stereoscopic image surfacess, the right multiplexing image Isr, and left multiplexed image Isl,and the composite multiplexed image Is. The right multiplexing mappingmpsr defines the spatial mapping of pixels in Ipr to pixels in Isr.Similarly, the left multiplexing mapping mpsl defines the spatialmapping of pixels in Ipl to pixels in Isl. The images Isr and Islrepresent the right and left eye stereoscopic perspectives of theobject(s) M. Isr and Isl are formed by the mappings mpsr and mpsl insuch a manner as to be compatible with the micro-polarizer basedstereoscopic image-pair display subsystem. Is is formed from Isr and Islas will be described later.

The stereoscopic image-pair display subsystem supports the mapping msd,the display surface sd, the right stereoscopic spatial multiplexeddisplay image Idr, the left stereoscopic spatial multiplexed displayimage Idl, and the composite stereoscopic spatial multiplexed displayimage Id, and the coordinate frame pd. The mapping msd defines themappings of the pixels of Is onto the display pixels as represented byId. The mapping msd represents an optical projection subsystem and caninclude some scaling factors between the image acquisition space RA andthe image display space RB. The images Idr and Idl form a realisticspatially multiplexed stereoscopic display image-pair which, when viewedby the stereoscopic image-pair viewing subsystem, form a realisticrepresentation, M', of the object(s) M. The virtual object M', isrepresented in the image display space, RB, which is referenced tocoordinate frame pw. The display surface sd, contains a micro-polarizerarray which performs polarization encoding of the images Idr and Idl.Surface sd has imbedded coordinates pd which are referenced to the imagedisplay space coordinates pw by the transformation Twd.

The stereoscopic image-pair viewing subsystem supports the right andleft optical imaging mappings mder and mdel, the right viewing surfaceser with imbedded coordinate system per, the left viewing surface selwith imbedded coordinate system pel, the right and left viewed imagesIer and Iel, the viewing coordinate system pv, and the visual processingsubsystem B. The right viewing image Ier is formed on the right viewingsurface ser by the right optical imaging mapping mder which performs apolarization decoding process. The left viewing image Iel is formed onthe left viewing surface sel by the left optical imaging mapping mdelwhich performs a polarization decoding process. The relationship betweenthe right an left viewing surfaces and the viewing coordinate system,pv, is given by the transformations Tver and Tvel respectively. Therelationship between the viewing coordinate system, pv, and the displaysurface coordinate system pd is given by the transformation Tdv. Thetransformations Tdv, Tver, and Tvel describe the position andorientation of the right and left viewing surfaces with respect to thedisplay surface sd.

FIG. 6B shows the process steps to be carried out when computing thetransformations and mappings to implement the system shown in FIG. 6A.The process steps are organized into five process groups, labeled Athrough E in FIG. 3b and are indicated by the object representationprocess steps (A), the stereoscopic image-pair acquisition process steps(B), the stereoscopic image multiplexing process steps (carried outusing spatial multiplexing) (C), the stereoscopic image-pair display(micro-polarization filter based) process steps (D), and thestereoscopic image-pair viewing process steps (based on polarizationdecoding processes) (E).

The object representation process steps operate on real physical object,Mr. The result of the object representation process steps is thecreation of an object representation M which is further processed by thestereoscopic image-pair acquisition process steps.

The stereoscopic image-pair generation process steps operate on theobject representation, M and produce the right and left pixelizedstereoscopic image-pairs Ipr and Ipl, respectively. The steps of thestereoscopic image-pair generation process use the transformations Tdv(acquired by the head position and orientation tracking subsystem), Tveland Tver (acquired by the eye position and orientation trackingsubsystem), and Twd (acquired by the display position and orientationtracking subsystem) and the acquisition of the display parameters msd,ss, and sd to compute various transformations and mappings as will bedescribed next.

The transformation Tmq describes the position and orientation placementof the right and left stereoscopic image-pair acquisition surfaces scrand scl. Tmq is computed by the function fTmq which accepts asparameters Twd, Tdv, Tvel, Tver, and pm. fTmq computes Tmq such that theimages Icr and Icl taken together form the beginnings of a stereoscopicimage-pair which represent a portion of the object(s) M.

The transformation Tqcr describes the position and orientation placementof the right stereoscopic image-pair acquisition surface Icr withrespect to pq. Tqcr is computed by the function fTqcr which accepts asparameters Tmq, Tdv, Tvel, and Tver. fTqcr computes Tqcr such that theimage Icr from the surface scr forms the beginnings of a realisticstereoscopic image-pair. In a similar manner, the transformation Tqcldescribes the position and orientation placement of the leftstereoscopic image-pair acquisition surface Icl with respect to pq. Tqclis computed by the function fTqcl which accepts as parameters Tmq, Tdv,Tvel, and Tver. fTqcl computes Tqcl such that the image Icl from thesurface scl forms the beginnings of a realistic stereoscopic image-pair.

The right object mapping, mmcr, creates an image representation, Icr, ofM, onto the surface scr. mmcr is computed by the function fmmcr whichaccepts as parameters Tmq, Tqcr, Tdv, Tver, sd, and msd. mmcr representsan optical imaging process (well known in the art). In a similar manner,the left object mapping, mmcl, creates an image representation, Icl, ofM, onto the surface scl. mmcl is computed by the function fmmcl whichaccepts as parameters Tmq, Tqcl, Tdv, Tvel, sd, and msd. mmcl representsan optical imaging process (well known in the art).

The image representation Icr on surface scr is formed by the functionfIcr which accepts as parameters mmcr and M. In a similar manner, theimage representation Icl on surface scl is formed by the function fIclwhich accepts as parameters mmcl and M. Mappings mmcr and mmcl aredefined in such a way that the images Icr and Icl taken together formthe beginnings of a realistic right and left, respectively, stereoscopicimage-pair which represents a portion of the object(s) M.

The right quantization mapping, mcpr, describes the conversion of theobject image Icr, into a pixelized image Ipr, on the synthetic surface,spr. mcpr is computed by the function fmcpr which accepts as parametersTdv, Tvel, Tver, sd, ss, and msd. The quantization mapping, mcprindicates how an arbitrary region of Icr is mapped into the pixelizedimage Ipr. In a similar manner, the left quantization mapping, mcpl,describes the conversion of the object image Icl, into a pixelized imageIpl, on the synthetic surface, spl. mcpl is computed by the functionfmcpl which accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd. Thequantization mapping, mcpl indicates how an arbitrary region of Icl ismapped into the pixelized image Ipr.

The right pixelized image Ipr on surface spr is formed by the functionfIpr which accepts as parameters Icr, mcpr, and dr, where dr representssupplemental pixel information. In a similar manor, the left pixelizedimage Ipl on surface spl is formed by the function fIpl which accepts asparameters Icl, mcpl, and dl, where dl represents supplemental pixelinformation. Mappings mcpr and mcpl are defined in such a way that theresulting images Ipr and Ipl will lead to the creation of a realisticstereoscopic image-pair in later steps of this process. Mappings mcprand mcpl can also be used to correct for limitations of an implementedsystem for performing the mappings mmcr and mmcl described above.

The stereoscopic image spatial multiplexing process steps operate on theright and left pixelized images Ipr and Ipl respectively and producesthe right and left spatially multiplexed stereoscopic imagerepresentations Isr and Isl. The steps of the stereoscopic image-pairspatial multiplexing process use the transformations Tdv (acquired bythe head position and orientation tracking subsystem), and Tvel and Tver(acquired by the eye position and orientation tracking subsystem), andthe acquisition of the display parameters msd, ss, and sd to computevarious transformations and mappings as will be described next.

The right multiplexing mapping, mpsr, defines the mapping of pixels inIpr to pixels in Isr. mpsr is computed by the function fmpsr whichaccepts as parameters Tdv, Tvel, Tver, sd, ss, and msd. In a similarmanner, the left multiplexing mapping, mpsl, defines the mapping ofpixels in Ipl to pixels in Isl. mpsl is computed by the function fmpslwhich accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd.

The right multiplexed image Isr, on surface ss, is formed by thefunction fIsr which accepts as parameters Ipr and mpsr. Likewise, theleft multiplexed image Isl, on surface ss, is formed by the functionfIsl which accepts as parameters Ipl and mpsl. Isr and Isl are formed bythe mappings mpsr and mpsl to be compatible with the micro-polarizingfilter based stereoscopic image-pair display subsystem. The compositemultiplexed stereoscopic image, Is, is formed from the compositing ofIsr and Isl.

The stereoscopic image-pair display process steps operate on the rightand left stereoscopic images Isr and Isl, respectively, using thedisplay mapping msd, to display the right and left stereoscopic imagedisplay pairs Idr and Idl on the micro-polarizer based display surface,sd. The mapping msd represent projection optics.

The right stereoscopic display image Idr, on surface sd, is formed bythe function/process fIdr which accepts as parameters Isr and msd.Likewise, the left stereoscopic display image Idl, on surface sd, isformed by the function/process fIdl which accepts as parameters Isl andmsd. The function/processes fIdr and fIdl form the stereoscopic encodingprocess which encodes the right and left stereoscopic display images,Idr and Idl, using polarized light (via the application of amicro-polarization panel 41 to the display surface sd) so as to beviewed in a stereoscopic viewing mode by the stereoscopic image-pairviewing process or processes. The composite multiplexed stereoscopicdisplay image, Id, is formed from the compositing of Idr and Idl.

The stereoscopic display surface, sd, has imbedded coordinates pd whichare related to pw by the transformation Twd. The display position andorientation tracking process tracks the interaction of the display withthe virtual environment M' and acquires the transformation Twd.

The stereoscopic image-pair viewing process steps represent the viewingdecoding of the right and left stereoscopic display images, Idr and Idl,through the decoding mappings mder and mdel, respectively, to producethe right and left viewer images, Ier and Iel, respectively. The rightand left viewer Images Ier and Iel are formed on the right and leftviewing surfaces ser and sel, respectively. The right viewing surface,ser, has imbedded coordinate frame per. Coordinate frame per is relatedto frame pv by the transformation Tver. Likewise, the left viewingsurface, sel, has imbedded coordinate frame pel. Coordinate frame pel isrelated to frame pv by the transformation Tvel. The function/processfIpr accepts parameters Idr, and mder and performs the actual decodingof the image Idr to form the image Ier using a polarizing filterdecoder. Likewise, the function/process fIpl accepts parameters Idl, andmdel and performs the actual decoding of the image Idl to form the imageIel using a polarizing filter decoder. The combination of images Ier andIel in the visual processing center, B, forms the image Ib. Ibrepresents the perceived stereoscopic image M' as represented in thevisual processing center B through the use of the function/process fIb.

Coordinate frame pv represents the imbedded coordinate frame of thecombined right and left viewing surfaces ser and sel, respectively. pvis related to the stereoscopic image-pair display coordinates system,pd, by the transformation Tdv.

The head position and orientation tracking process tracks theinteraction of the combined right and left viewing surfaces, ser andsel, with the display surface, sd, and acquires the transformation Tdvto describe this interaction. The eye position and orientation trackingprocess tracks the interaction of each individual right and left viewingsurface, ser and sel, with respect to the coordinate frame pv, andacquires the right and left viewing surface transformations, Tver andTvel.

The overall process steps defined by the process groups A through E inFIG. 6B define interactive process which starts with the acquisition ofthe transformations Tdv (acquired by the head position and orientationtracking subsystem), Tvel and Tver (acquired by the eye position andorientation tracking subsystem), and Twd (acquired by the displayposition and orientation tracking subsystem) and the acquisition of thedisplay parameters msd, ss, and sd. The interactive process continueswith the process steps A, B, C, D, and E in the given order and thenrepeats with the acquisition of the above transformations andparameters.

Having described the illustrative embodiments of the present invention,several modifications readily come to mind.

In particular, the display subsystem 6 and display surface 40 mayberealized using the 3-D projection display system and surface disclosedin Applicants copending application Ser. No. 08/339,986.

The system and method of the present invention have been described ingreat detail with reference to the above illustrative embodiments.However, it is understood that other modifications to the illustrativeembodiments will readily occur to persons with ordinary skill in theart. All such modifications and variations are deemed to be within thescope and spirit of the present invention as defined by the accompanyingClaims to Invention.

                  APPENDIX A                                                      ______________________________________                                        Symbols/Operators                                                             ______________________________________                                        *       Multiplex Images (temporal, spatial, spectral)                        M       Model or Objects (virtual or physical)                                px      Coordinate system x                                                   Rx      Space x (virtual or physical)                                         Txy     Transformation between coordinate frames x and y                      Ix      Image representation x (geometric or pixelated)                       sx      Surface of image representation x (shown flat but does not                    need to be)                                                           dx      Image data                                                            ______________________________________                                    

                  APPENDIX B                                                      ______________________________________                                        Symbols in Figure Drawings                                                    ______________________________________                                        RA     Model space (real or virtual)                                          RB     Viewing space (virtual)                                                M      Model (real or synthetic, M = (Mr or Ms))                              Mr     Real Model                                                             Ms     Virtual/Synthetic Model                                                M'     Representation of model by stereoscopic display(virtual)               B      Visual processing center                                               pm     Coordinate of model                                                    pq     Coordinate of stereoscopic image generation/capture device             pcl, pcr                                                                             Coordinates of left and right image generation view surfaces           pw     Coordinate of virtual representation of model                          pd     Coordinate of stereoscopic image display surface                       pv     Coordinate of stereoscopic viewing system                              pel, per                                                                             Coordinates of left and right viewing view surfaces                    Tmq    Transformation from pm to pq                                           Tqcl, Tqcr                                                                           Transformation from pq to pcl and from pq to pcr                       Twd    Transformation from pw to pd                                           Tdv    Transformation from pd to pv                                           Tvel, Tver                                                                           Transformation from pv to pel and from pv to per                       Icl, Icr                                                                             Left and right geometric model image representation                    Ipl, Ipr                                                                             Left and right pixelized image representations of Icl and Icr          Is     Stereoscopic multiplexed (time, spatial, frequency, etc.) image               representation of Ipl and Ipr                                          ______________________________________                                    

                  APPENDIX C                                                      ______________________________________                                        Isl, Isr                                                                              Left and right components of Is (for reference)                       Id      Is as represented on a stereoscopic display surface                   Idl, Idr                                                                              Left and right components of Id                                       Iel, Ier                                                                              Left and right images representations on the viewing view                     surfaces                                                              Ib      Perceived image in the vision processing center B                     dl, dr  Supplemental left and right display information                       scl, scr                                                                              Left and right geometric model image representation                           surfaces (size and aspect ratio information)                          spl, spr                                                                              Left and right pixelized image representations surfaces                       (resolution and aspect ratio information)                             ss      Stereoscopic multiplexed image representation surface                         (resolution and aspect information)                                   sd      stereoscopic display surface (size and aspect information)            sel, ser                                                                              Left and right image viewing view surfaces                            mmcr, mmcl                                                                            Optical or virtual geometric mapping from M to Icl and Icr            mcpl, mcpr                                                                            Quantization and pixel mapping from Icl to Ipl and Icr                        to Ipr                                                                mpsl, mpsr                                                                            Multiplexing process to convert Ipl and Ipr into the                          stereoscopic image Is                                                 msd     Mapping from Is to the display image Id (determined by the                    projection/display system and optics) msd determines the                      transformation between the scale of the image generation                      space and the viewing and display spaces.                             mder, mdel                                                                            Mapping from Id to the viewing system representations                         Iel and Ier                                                           ______________________________________                                    

What is claimed is:
 1. A system for producing and displayingstereoscopically-multiplexed images of either real or synthetic 3-Dobjects presented or represented in an object space in a way whichpermits realistic stereoscopic viewing thereof in a virtual reality (VR)viewing environment, said system comprising:image display means disposedwithin an image display space, and having a display surface fordisplaying said stereoscopically-multiplexed images of said real orsynthetic 3-D objects; parameter acquisition means for continuouslyacquiring a complete set of viewing parameters of a viewer disposedrelative to said display surface, said complete set of viewingparameters includinginformation regarding the position and orientationof the eyes of the viewer relative to a first coordinate reference frameembedded within the head of said viewer, information regarding theposition and orientation of said head relative to a second coordinatereference frame embedded within said display surface, and informationregarding the position and orientation of said image display meansrelative to a third coordinate reference frame embedded within saidimage display space; stereoscopic image pair production means forproducing stereoscopic image pairs of said real or synthetic 3-D objectsusing said complete set of viewing parameters;stereoscopically-multiplexed image production means for producing saidstereoscopically-multiplexed images of said real or synthetic 3-Dobjects using said stereoscopic image pairs; image mapping means formapping said stereoscopic image pairs onto said display surface forviewing by said viewer; and image viewing means through which saiddisplayed stereoscopically-multiplexed images can be viewed by the eyesof said viewer so as to permit realistic stereoscopic viewing of saidreal or synthetic 3-D objects in said VR viewing environment.
 2. Thesystem of claim 1, wherein said VR viewing environment is realized as aflight simulation and training environment, a telerobotic environment, avirtual surgery environment, or a video-gaming environment.
 3. Thesystem of claim 2, wherein said virtual surgery environment is alaparascopic or endoscopic surgical environment.
 4. The system of claim1, wherein said stereoscopically-multiplexed images arespatially-multiplexed images (SMIs) of said real or synthetic 3-Dobjects, and wherein said image display means comprises either anelectrically passive micropolarization panel or an electronicallyreprogrammable micropolarization panel.
 5. The system of claim 1,wherein said image viewing means comprises a pair ofelectrically-passive polarizing eye-glasses.
 6. The system of claim 1,realized in the form of desktop computer graphics workstation, adaptedfor use in virtual reality applications.
 7. The system of claim 1,wherein said stereoscopically-multiplexed image production meanscarrying out either a time-sequential multiplexing process, aspatial-multiplexing process, or a spectral-multiplexing process.
 8. Thesystem of claim 1, wherein said image display means comprises an imageprojection subsystem.
 9. The system of claim 8, wherein said imageprojection subsystem is mounted onto a moveable support platform for usein flight-simulators, virtual-reality games.
 10. The system of claim 1,which further comprisesobject representation means for representing saidsynthetic object in said object space, so that said stereoscopic imagepair production means produces said stereoscopic image pairs of saidreal or synthetic 3-D objects using said complete set of viewingparameters obtained from said parameter acquisition means.
 11. Astereoscopic image display subsystem for displaying and viewing imagesof virtual 3-D objects represented in image display space, comprising:adisplay surface having a first coordinate reference frame symbolicallyembedded therein; a head/eye position and orientation tracking subsystemincludingminiature head/eye imaging cameras for capturing images of aviewer's eyes, an image processing computer for processing said capturedimages on a real-time basis and producing a first information setrepresentative of position and orientation of the eyes and head of saidviewer relative to said first coordinate reference frame; and a displaysurface position and orientation tracking subsystem, operably associateswith said image processing computer, for producing on a real-time basis,a second information set representative of the position and orientationof said display surface relative to a second coordinate reference framesymbolically embedded within said image display space.
 12. Thestereoscopic image display subsystem of claim 11, which furthercomprises:first information processing means for processing said firstand second information sets and producing a first complete set ofcoordinate frame transformations which relate left and right perspectiveimages seen by said viewer, to said second coordinate reference framewith respect to which a geometrical structure of said virtual 3-Dobjects is referenced.
 13. The stereoscopic image display subsystem ofclaim 12, in combination with a stereoscopic image-pair generationsubsystem which comprises:a pair of object projection surfaces uponwhich images of said 3-D objects are projected according to left andright perspective image mapping processes, respectively, in order toproduce said left and right perspective images; second informationprocessing means for processing said first complete set of coordinateframe transformations and producing a second complete set of coordinateframe transformations; and mapping process parameterization means forparameterizing said left and right perspective image mapping processessuch that said left and right perspective images seen by said viewer aresubstantially similar to the left and right perspective images projectedonto said pair of object projection surfaces.
 14. An interactive-basedsystem for producing and displaying stereoscopically-multiplexed images,comprising:(A) object representation subsystem for representing anobject M relative to a first coordinate reference frame pm; (B) astereoscopic image display subsystem including(1) a display structurehaving a display surface positioned relative to a viewer and supportingan array of pixels located at a set of coordinates specified by acoordinate set sd referenced relative to a second coordinate referenceframe pd symbolically embedded in said display structure, said displaysurface being provided for displaying stereoscopically-multiplexedimages Is of a virtual object M' corresponding to the object M, andbeing referenced relative to a third coordinate reference frame pw sothat the eyes in the head of said viewer can view saidstereoscopically-multiplexed images Is displayed on said displaysurface; (2) an eye/head position and orientation tracking subsystemincluding means for tracking the position and orientation of the headand eyes of the viewer relative to said display surface and producinghead and eye position and orientation parameters, means for producing aleft eye viewing transformation Tvel parameterized by said head and eyeposition and orientation parameters transformed to said third coordinatereference frame pw, means for producing a right eye viewingtransformation Tver parameterized by said head and eye position andorientation parameters transformed to said third coordinate referenceframe pw, and means for producing a head-display viewing transformationTdv parameterized by said head and eye position and orientationparameters transformed to said third coordinate reference frame pw; (3)a display surface position and orientation tracking subsystem includingmeans for tracking the position and orientation of said display surfacerelative to said third coordinate reference frame pw and producingdisplay surface position and orientation parameters, means for producinga head/eye viewing transformation Twd parameterized by said displaysurface position and orientation parameters transformed to said secondcoordinate reference frame pd and relating said second coordinatereference frame pd to said third coordinate reference frame pw, meansfor providing a set of display surface parameters sd representing thesize and aspect ratio of said display surface, and means for providing aset of image generation-display mapping parameters msd representing aprocess of mapping the pixels in the stereoscopically-multiplexed imageIs onto pixels of a mapped stereoscopically-multiplexed image Id beingdisplayed on said display surface; (C) stereoscopic image-pairproduction subsystem for producing stereoscopic image-pairs, consistingleft and right perspective images of said object M, includingmeans forgeometrically defining said object M relative to said first coordinatereference frame pm, means for providing a left image acquisition surfacescl on which said left perspective image of said object M is acquiredand referenced relative to a fourth coordinate reference frame pcl, anda right image acquisition surface scr on which a right perspective imageof said object M is acquired and referenced relative to a fifthcoordinate reference frame pcr, wherein said left and right objectsurfaces scl and scr being defined relative to a sixth coordinatereference frame pq, means for producing an object-camera viewingtransformation Tmq parameterized by said transformations Twd, Tdv, Tvel,Tver, and said first coordinate reference frame pm, for specifying theposition and orientation placement of said left and right imageacquisition surfaces scl and scr relative to said first coordinatereference frame pm, means for producing a left image acquisition surfacetransformation Tqcl parameterized by said Twd, Tdv, Tvel, and Tver, fordescribing the position and orientation placement of the left imageacquisition surface scl relative to said sixth coordinate referenceframe pq, means for producing a right image acquisition surfacetransformation Tqcr parameterized by said Twd, Tdv, Tvel and Tver, fordescribing the position and orientation placement of the right imageacquisition surface scr relative to said sixth coordinate referenceframe pq, means for mapping said left perspective image of said object Monto said left image acquisition surface scl using, as parameters, saidtransformations Tmq, Tqcl, Tdv and Tvel, and said parameters sd and msd,and means for mapping said right perspective image of said object M ontosaid right image acquisition surface using, as parameters, saidtransformations Tmq, Tqcr, Tdv and Tver, and said parameters sd and msd;and (D) stereoscopically-multiplexed image subsystem includingmeans forproducing a stereoscopically multiplexed image from each of saidstereoscopic image-pairs, for mapping onto said display surface by wayof said mapping process, and means for producingstereoscopically-multiplexed image parameters ss for representing theresolution and aspect ratio of the stereoscopically multiplexed image.15. The system of claim 14, wherein said left and right imageacquisition surfaces are associated with a real camera located withrespective to a real object in object space R_(A).
 16. The system ofclaim 15, wherein said left and right image acquisition surfaces areassociated with a virtual camera located with respective to a virtual(computer-represented) object in image display space R_(B).
 17. Anprocess for producing and displaying stereoscopically-multiplexedimages, comprising the steps:(1A) representing an object M relative to afirst coordinate reference frame pm; (1B) providing a stereoscopic imagedisplay subsystem including a display structure having a display surfacepositioned relative to a viewer and supporting an array of pixelslocated at a set of coordinates specified by a coordinate set sdreferenced relative to a second coordinate reference frame pdsymbolically embedded in said display structure, said display surfacebeing provided for displaying stereoscopically-multiplexed images Is ofa virtual object M' corresponding to said object M, and being referencedrelative to a third coordinate reference frame pw so that the eyes inthe head of said viewer can view said stereoscopically-multiplexedimages Is displayed on said display surface; (2A) tracking the positionand orientation of the head and eyes of the viewer relative to saiddisplay surface and producing head and eye position and orientationparameters; (2B) producing a left eye viewing transformation Tvelparameterized by said head and eye position and orientation parameterstransformed to said third coordinate reference frame pw; (2C) producinga right eye viewing transformation Tver parameterized by said head andeye position and orientation parameters transformed to said thirdcoordinate reference frame pw; and (2D) producing a head-display viewingtransformation Tdv parameterized by said head and eye position andorientation parameters transformed to said third coordinate referenceframe pw; (3A) tracking the position and orientation of said displaysurface relative to said third coordinate reference frame pw andproducing display surface position and orientation parameters; (3B)producing a head/eye viewing transformation Twd parameterized by saiddisplay surface position and orientation parameters transformed to saidsecond coordinate reference frame pd and relating said second coordinatereference frame pd to said third coordinate reference frame pw; (3C)providing a set of display surface parameters sd representing the sizeand aspect ratio of said display surface; and (3D) providing a set ofimage generation-display mapping parameters msd representing a processof mapping the pixels in the stereoscopically-multiplexed image Is ontopixels of a mapped stereoscopically-multiplexed image Id being displayedon said display surface; (4A) geometrically defining said object Mrelative to said first coordinate reference frame pm; (4B) providing aleft image acquisition surface scl on which a left perspective image ofsaid object M is acquired and referenced relative to a fourth coordinatereference frame pcl, and a right image acquisition surface scr on whicha right perspective image of said object M is acquired and referencedrelative to a fifth coordinate reference frame pcr, wherein said leftand right object surfaces scl and scr being defined relative to a sixthcoordinate reference frame pq; (4C) producing an object-camera viewingtransformation Tmq parameterized by said transformations Twd, Tdv, Tvel,Tver, and said first coordinate reference frame pm, for specifying theposition and orientation placement of said left and right imageacquisition surfaces scr and scl relative to said first coordinatereference frame pm; (4D) producing a left image acquisition surfacetransformation Tqcl parameterized by said Twd, Tdv, Tvel, and Tver, fordescribing the position and orientation placement of the left imageacquisition surface scl relative to said sixth coordinate referenceframe pq; (4E) producing a right image acquisition surfacetransformation Tqcr parameterized by said Twd, Tdv, Tvel and Tver, fordescribing the position and orientation placement of the right imageacquisition surface relative scr to said sixth coordinate referenceframe pq; (4F) mapping said left perspective image of said object M ontosaid left image acquisition surface scl using, as parameters, saidtransformations Tmq, Tqcl, Tdv and Tvel, and said parameters sd and msd;and (4G) mapping said right perspective image of said object M onto saidright image acquisition surface scr using, as parameters, saidtransformations Tmq, Tqcr, Tdv and Tver, and said parameters sd and msd;and (5A) producing a stereoscopically multiplexed image from said leftand right perspective images mapped onto said left and right imageacquisition surfaces scl and scr, respectively, for mapping onto saiddisplay surface by way of said mapping process; and (5B) producingstereoscopically-multiplexed image parameters ss for representing theresolution and aspect ratio of the stereoscopically multiplexedimage(s).
 18. The process of claim 17, wherein said left and right imageacquisition surfaces are associated with a real camera located withrespective to a real object in object space R_(A).
 19. The process ofclaim 17, wherein said left and right image acquisition surfaces areassociated with a virtual camera located with respective to a virtual(computer-represented) object in image display space RB.